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ABSTRACT 





The goal of this project is to provide a Human Computer Interaction system to resolve the problem faced 
by the deaf and dumb people. Since the algorithm is not designed on the base of background hand 
gestures, it is immune to changes in the background image. It can process a variety of hand types, identify 
a number of fingers, and perform tasks as needed. The key objectives were met, as stated in this paper. 
Real-time gesture recognition is possible with this programme. There are certain obstacles that must be 
addressed in the future. For human-computer interaction, hand gesture recognition is critical. The hand 
region is extracted from the context in our system using background subtraction. The palm and fingers are 
then segmented so that the fingers can be detected and recognised. Finally, a rule classifier is used to 
predict hand gesture labels. Experiments on a 1300 image data set show that our method works well and 
is very effective. Furthermore, on another data collection of hand movements, our approach outperforms 
a method called state of the art. Gesture recognition is one of the essential techniques to build user- 
friendly interfaces. 

Keywords: Human Computer Interaction System, Gesture Recognition, Data Set. 





nurses to arising issues in_ patient care. 


1. Introduction Accelerometers and gyroscopes are used in many 


Smart devices have proliferated due to the low 
costs and small sizes of new digital technology. 
People now have more smart phones in their hands 
than ever before, and there is more data available 
than ever before. One solution, known as the 
Internet of Things (IoT), aims to link these 
embedded devices to automatically send and 
receive data, enabling us to enhance quality of life 
through data application. The Internet of Things 
has been widely used in the medical sector, 
ranging from basic self-tracking of sleep and heart 
rate to weight sensing and redistribution for 
overweight patients to avoid ulcers, as well as 
monitoring vital signs in hospitals and alerting 
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IoT devices, allowing for the measurement of 3- 
dimensional linear accelerations and angular 
velocities. The use of these sensors to detect 
individual human movements, such as smoking 
and opening a bottle of medicine, to alter habits 
that contribute to negative health consequences is 
one of the many applications of these sensors. 
Human motion recognition is a hotly debated 
subject with numerous challenges. Inside each 
movement, dynamic variation exists, allowing for 
slow or quick movements, as well as exaggerated 
or subtle movements. Another difficulty is 
extrapolating data from a single test subject to 
multiple users, who may have minor variations in 
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their individual movements. Computers have 

become an integral part of life in today's world, 

and they are used in a variety of fields. However, 
the systems and techniques we use to communicate 

with computers are obsolete, and they have a 

number of problems, which we will address later in 

this article. As a result, a brand-new area called 

Human Computer Interactions has arisen to 

address these issues (HCI). [1-5] 

2. Problem Identification 

2.1. Aim of the paper 

The Paper mainly focuses to reduce the problem 

faced by the deaf and dumb people. The sign 

language recognition system helps the people to 
easily communicate with the normal people 
without any interruption. The system aims to 
recognize the sign language and translate it to the 
local language via text. Thus the problem between 
deaf and dumb people and normal people can be 
solved using this system. A Hand Gesture 

Recognition System (HGRS) recognizes shapes 

and orientations, depending on the implementation, 

in order to task the system to perform a task. 

Gestures are a form of nonverbal communication. 

At any given time, a person may make multiple 

gestures. Since humans _ interpret human 

movements through vision and computers need a 

camera, computer vision research is a hot topic. 

2.2 Objective of the Project 

The project's goals are to: 

e Research and apply the necessary tools, which 
include: 

o a_ robot with unidirectional movement 
Computer vision and artificial intelligence 
algorithms 

o The OpenCV Computer Vision Library 

e Create a computer vision programme that 
recognises basic gestures. 

e Examine the computer programme 

e Keep track of the project's outcomes. 

3. Pattern recognition 

3.1. Computer Vision 

A real object in computer vision maps to a 

segmented region in the image from which object 

descriptors or features can be extracted. Any 
characteristic of an image or any region within it 
that can be calculated is referred to as a function. 

Objects with similar characteristics can be 

categorised into groups, and the combination of 

characteristics can be called as pattern. The 
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creation of formal descriptors is the most 
important step because it requires decisions about 
which characteristics, quantitative or qualitative, 
will better fit the target product, which determines 
the success of the project. Quantitative 
descriptions known as features are used in 
statistical pattern recognition. The pattern vector or 
function vector is made up of the collection of 
features, and the pattern space X is made up of all 
possible patterns for the object (also known as 
feature space). Quantitatively, similar objects in 
each class may cluster together in the feature 
space, which will hopefully be useful. Classifier 
architecture includes determining the best 
discrimination feature or Discriminant to use. A 
statistical classifier takes n features as inputs and 
produces one output: a classification or judgement 
about the object's class. A decision rule, which is a 
function that places certain feature vectors in one 
space or subset, determines the relationship 
between the inputs and the outputs. The decision 
rule is based on the specific discrimination 
mechanism that was used to separate the subsets. 
Classifier learning is the ability of a classifier to 
classify objects based on its decision rule, and the 
training set is the set of feature vectors (objects) 
inputs and corresponding classification outputs 
(both positive and negative results).[6-10]. 

3.2. Special Role of Heuristic Rules 

If pattern recognition is accomplished using 
complex and advanced artificial intelligence 
methods, heuristic principles have a place in 
computer vision. A heuristic is empirical yet 
ambiguous intelligence. Finding an exact and 
exhaustive solution to a problem of high 
complexity is often impractical or prohibitively 
costly, if not impossible. There are periods when 
no algorithm exists or cannot be found to provide a 
solution. Sometimes, the purpose is merely 
exploratory, and all that is needed for the time 
being is an acceptable estimation or approximation 
of a potential solution. In these cases, all that is 
needed is a good heuristic or rule of thumb to 
arrive at a good working solution. 

4. System Design 

4.1. Algorithm for Colour Segmentation 
Thresholding is a technique that is used to limit the 
amount of data that is the process of defining 
regions within an image is known as segmentation. 
Segmentation may be aided by the use of colour. 
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The region of interest in this project was the hand 
on the picture. The range of HSV values for skin 
colour was calculated for use as the threshold 
values to separate the picture pixels of the hand 
from the background. After converting all pixels 
within those threshold values to white and those 
without to black, segmentation could begin. 

By using thresholding the algorithm for colour 
segmentation: 

1. Use the camera to capture a picture of the 
gesture. 

2. Determine the HSV value set for skin colour 
that will be used as threshold values. 

3. Change the colour space of the image from 
RGB to HSV. 

4. Convert to white all pixels that fall below the 
threshold values. 

5. Make other pixels black. 

6. Image segmented should be saved in image file. 





Fig.1.Image comparison 
Before interpreted by colour segmented image the 
algorithm Labelling and Blob detection is used, it 
must be recognised as a single entity. This can be 
accomplished via the marking and blob detection 
processes.. The adjacency relationships between 
pixels can be used to determine if they belong to 
the same area. The following are the two most 
common adjacency relationships: 
° 4-adjacency pair 
¢ 8-neighbourhood 
A pixel is bound to its neighbouring pixels in 4- 
adjacency if they occupy the leftmost, top-most, 
right-most, and bottom positions with respect to 
the pixel. A four-adjacency relationship for pixels 
was created using the (x,y) coordinate descriptions 
(x, y). The top-left-most, top-right-most, bottom- 
leftmost, and bottom-right-most pixels are all 
adjacent pixels in 8-adjacency. A 8-adjacency 
relationship for pixels was created using the (x,y) 
coordinate descriptions (x, y). 
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4.2. Data Obtaining and Pre-Processing 

Data collection: The image is captured first for 
identifying the frame. This is crucial because the 
image can contain a number of variables, and these 
variables can cause unintended effects, reducing 
the data that needs to be processed. A webcam is 
used to capture the image, which continuously 
captures frames and provides raw data for 
processing. The uint8 input image is what we have 
here. The procured image is RGB must be 
processed before the components are separated and 
recognition is rendered, i.e. pre-processed. 
Pre-processing of data: The pre-processing 
method is a two-step process: * Customer 
segmentation Filtering by morphology The 
Segmentation process is the first step. It is done to 
convert a grey-scale image to a binary image, 
allowing us to have only two areas of interest in 
the image. One will be the side, and the other will 
be the background. For this method, the Otsu 
algorithm is used, and a greyscale image is 
transformed into a binary image with the hand as 
the area of interest. 

Segmentation can be done in one of two ways: 

1. Local or pixel-based methods that include: 

a) Detecting the Edge 

b) Identification of boundaries 

2. A region-based approach includes: 

a) Mixing the region, 

b) Splitting the region, and 

c) The threshold process. 

4.3. Android Application 

The system is developed for the purpose to help 
the deaf and dumb people .The application is 
developed using Python and HTML, CSS by inter 
crossing the backend results to a_ successful 
application .By using image array the background 
images are not taken for recognition as it reflects 
the hand gestures to wrong view ,which it turn 
reflects the sign language in wrong way. 


Conclusion 

This paper introduces a new approach for hand 
gesture recognition. The background subtraction 
method is used to detect the hand region from the 
background. The palm and fingers are segmented 
after that. The fingers in the hand picture are 
discovered and recognised based on_ the 
segmentation. Hand gesture recognition is 
achieved using a basic rule classifier. On a data set 
of 1300 hand pictures, the output of our system is 
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evaluated. The results of the experiments show 

that our method works well and is suitable for real- 

time applications. Furthermore, on a picture series 
of hand movements, the proposed approach 
outperforms the state-of-the-art FEMD 
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